Skip to content

STAC-25500 Port the DEB signing and pre-release publishing lane to GitHub Actions - #455

Merged
LouisParkin merged 3 commits into
STAC-25457-publish-lanefrom
STAC-25500-deb-prerelease
Aug 10, 2026
Merged

STAC-25500 Port the DEB signing and pre-release publishing lane to GitHub Actions#455
LouisParkin merged 3 commits into
STAC-25457-publish-lanefrom
STAC-25500-deb-prerelease

Conversation

@LouisParkin

Copy link
Copy Markdown

Closes the last two GitLab jobs on this repo with no GitHub counterpart: sign_deb and pre_release_deb (.gitlab-ci-agent.yml lines 604 and 635). Stacked on #454.

Security prerequisite, not cleanup

sign_debian_package.sh called printenv unconditionally. On GitLab that dumped the GPG private key and its passphrase into the job log. stackstate-agent is a public repo, so on GitHub that log is world-readable. It also left the exported private key in the checkout as gpg_private.key, where any later artifact upload would collect it.

Both are removed before any signing secret is wired in, as STAC-25500 requires. Key setup now happens in an ephemeral GNUPGHOME (mktemp, mode 700) removed by an EXIT trap that also kills the gpg-agent, so a preset passphrase cannot outlive the job on a reused runner. The passphrase reaches gpg through a mode-600 file, never argv.

Fixes a latent signing bug

The old preset step was:

echo $SIGNING_PRIVATE_PASSPHRASE | gpg-preset-passphrase -v -c $(gpg --list-secret-keys ... | awk ...)

A key exposes one keygrip per primary and subkey. Unquoted, that substitution expands to multiple words, so with a signing subkey the second keygrip became a stray argument and was never presetted — leaving signing able to block on a pinentry prompt no CI runner can answer. Each keygrip is now presetted individually.

Verified against a generated throwaway key: two keygrips, both presetted, passphrase delivered on stdin, GNUPGHOME removed by the trap, nothing left in the workspace.

Shared setup

The two scripts need the same key in the same state but are separate processes, and on GitHub may be separate steps, so neither can rely on a keyring the other left behind. The setup moves into gpg_signing_setup.sh, sourced by both. gpg-preset-passphrase is auto-detected across /usr/lib/gnupg2, /usr/lib/gnupg, /usr/libexec and PATH (the path differs by distro and release), and it fails loudly rather than silently skipping the preset when absent.

Both scripts fall back CI_PROJECT_DIRGITHUB_WORKSPACE, and publish_package.sh falls back CI_COMMIT_REF_NAMEGITHUB_REF_NAME, so the apt codename stays the release branch exactly as on GitLab.

Workflow

Job Notes
sign-and-publish-deb Downloads both arch artifacts and runs a single deb-s3 upload. GitLab fanned this out per arch, so two jobs rewrote the same apt index concurrently — collecting both first removes that race.
generate-install-script Runs inv release.generate-install -t inside the build container: the task collection imports python-gitlab plus the GitHub/Datadog API helpers and only loads in the conda env. Asserts the rendered script contains no None.s3.amazonaws.com from an unset bucket variable — generate_install reads three bucket vars and silently renders None for any that are missing.
publish-install-script Uploads it, so the container job never holds a credential.

Both publishing jobs are gated on push and bound to the agent-pre-release environment, which carries the deployment branch rule, the signing secrets and the AWS role. Pull requests cannot reach them.

Blocked on

This lane cannot go green until these exist — the code is ready and reviewable now:

  1. GitHub environment agent-pre-release with a deployment branch rule for the release branch.
  2. Secrets in it: SIGNING_PUBLIC_KEY, SIGNING_PRIVATE_KEY, SIGNING_PRIVATE_PASSPHRASE, SIGNING_KEY_ID.
  3. Variable AGENT_PRERELEASE_ROLE_ARN, from StackVista/terraform-infra#76 (STAC-25545).

Validation

  • shellcheck -x — clean on all three scripts.
  • gpg_signing_setup exercised against a real generated key: explicit override, PATH auto-detection, and the missing-binary failure path (exits 1 with a clear message).
  • actionlint — clean apart from the pre-existing self-hosted runner-label notices.
  • zizmor v1.28.0 — No findings to report.
  • YAML parses; all 10 needs: references resolve.

https://stackstate.atlassian.net/browse/STAC-25500
Refs https://stackstate.atlassian.net/browse/STAC-25546 and https://stackstate.atlassian.net/browse/STAC-25545

Comment thread .github/workflows/build-deb.yml Outdated
Comment thread .github/workflows/build-deb.yml Outdated
LouisParkin added a commit that referenced this pull request Aug 6, 2026
…tion

Addresses review feedback on PR #455.

[P1] The signing job installed the publisher with `gem install deb-s3`,
resolving the latest code at run time into a job that then executes it with
the package signing key and the pre-release AWS credentials in scope. GitLab
never did this: `sign_deb` ran in a pinned image with deb-s3 already baked in,
so the runtime resolve was a regression introduced by the port.

deb-s3 and its full runtime dependency tree are now pinned in
`.github/deb-s3-gems.sha256` and installed by `.github/scripts/install-deb-s3.sh`,
which fetches each gem at its exact version and verifies it against the SHA256
RubyGems publishes for that release before anything is installed or executed.

The manifest covers nine gems. base64, bigdecimal and logger are deliberately
excluded: aws-sdk-core requires them at ">= 0" and they are Ruby default gems
supplied by the distribution's own ruby package, so pinning them would force a
native build for no supply-chain gain.

The script also links the canonical executable when RubyGems installs a
versioned binstub, since publish_package.sh invokes deb-s3 by bare name, and
ends with a `deb-s3 help` smoke check that activates the whole pinned set so a
missing or incompatible gem fails at install time rather than midway through
publishing.

[P2] publish-install-script depended only on generate-install-script, so it
could overwrite the public install.sh even when the build, signing or apt
upload had failed. GitLab's pre_release_deb required sign_deb. It now requires
both generate-install-script and sign-and-publish-deb, restoring that release
boundary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LouisParkin added a commit that referenced this pull request Aug 6, 2026
…tion

Addresses review feedback on PR #455.

[P1] The signing job installed the publisher with `gem install deb-s3`,
resolving the latest code at run time into a job that then executes it with
the package signing key and the pre-release AWS credentials in scope. GitLab
never did this: `sign_deb` ran in a pinned image with deb-s3 already baked in,
so the runtime resolve was a regression introduced by the port.

deb-s3 and its full runtime dependency tree are now pinned in
`.github/deb-s3-gems.sha256` and installed by `.github/scripts/install-deb-s3.sh`,
which fetches each gem at its exact version and verifies it against the SHA256
RubyGems publishes for that release before anything is installed or executed.

The manifest covers nine gems. base64, bigdecimal and logger are deliberately
excluded: aws-sdk-core requires them at ">= 0" and they are Ruby default gems
supplied by the distribution's own ruby package, so pinning them would force a
native build for no supply-chain gain.

The script also links the canonical executable when RubyGems installs a
versioned binstub, since publish_package.sh invokes deb-s3 by bare name, and
ends with a `deb-s3 help` smoke check that activates the whole pinned set so a
missing or incompatible gem fails at install time rather than midway through
publishing.

[P2] publish-install-script depended only on generate-install-script, so it
could overwrite the public install.sh even when the build, signing or apt
upload had failed. GitLab's pre_release_deb required sign_deb. It now requires
both generate-install-script and sign-and-publish-deb, restoring that release
boundary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@LouisParkin
LouisParkin force-pushed the STAC-25500-deb-prerelease branch from f31b3c3 to ed3b944 Compare August 6, 2026 14:39
@LouisParkin
LouisParkin force-pushed the STAC-25457-publish-lane branch from 2d5cbce to 61e2bce Compare August 7, 2026 09:26
LouisParkin added a commit that referenced this pull request Aug 7, 2026
…tion

Addresses review feedback on PR #455.

[P1] The signing job installed the publisher with `gem install deb-s3`,
resolving the latest code at run time into a job that then executes it with
the package signing key and the pre-release AWS credentials in scope. GitLab
never did this: `sign_deb` ran in a pinned image with deb-s3 already baked in,
so the runtime resolve was a regression introduced by the port.

deb-s3 and its full runtime dependency tree are now pinned in
`.github/deb-s3-gems.sha256` and installed by `.github/scripts/install-deb-s3.sh`,
which fetches each gem at its exact version and verifies it against the SHA256
RubyGems publishes for that release before anything is installed or executed.

The manifest covers nine gems. base64, bigdecimal and logger are deliberately
excluded: aws-sdk-core requires them at ">= 0" and they are Ruby default gems
supplied by the distribution's own ruby package, so pinning them would force a
native build for no supply-chain gain.

The script also links the canonical executable when RubyGems installs a
versioned binstub, since publish_package.sh invokes deb-s3 by bare name, and
ends with a `deb-s3 help` smoke check that activates the whole pinned set so a
missing or incompatible gem fails at install time rather than midway through
publishing.

[P2] publish-install-script depended only on generate-install-script, so it
could overwrite the public install.sh even when the build, signing or apt
upload had failed. GitLab's pre_release_deb required sign_deb. It now requires
both generate-install-script and sign-and-publish-deb, restoring that release
boundary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@LouisParkin
LouisParkin force-pushed the STAC-25500-deb-prerelease branch from ed3b944 to e3a578b Compare August 7, 2026 09:28
@LouisParkin

Copy link
Copy Markdown
Author

Rebase regression fixed — amd64 gate is green

1a2d643ef3 restores two lines that the earlier rebase silently reverted.

When this branch was rebased onto the updated #454, the conflict in build-deb.yml was resolved with git checkout --theirs, which takes the replayed commit's whole copy of the file. That copy predated the exception work on #454, so it quietly dropped the quay-qualified LOCAL_IMAGE and exceptions-path: exceptions. It passed lint and read fine — the only symptom was in the gate summary.

Before (e3a578b, amd64):

total in-scope findings:  24
suppressed by exception:  0
unmanaged:                24

After (1a2d643, amd64):

total in-scope findings:  24
suppressed by exception:  24
unmanaged:                0

That now matches #454 exactly. Verified structurally too — git diff STAC-25457-publish-lane -- .github/workflows/ on this branch is additions-only, no removed lines.

Remaining red is one external gap, not this PR

The only failing jobs left on this branch are the two arm64 container image builds:

  • Build agent container image (docker build, no push on PR, arm64)
  • Build cluster-agent container image (docker build, no push on PR, arm64)

Both die inside scan-image with:

##[error]VEX repositories downloaded, but no OpenVEX documents were found for Grype

The message is misleading. The real cause is that the lean arm64 runner image does not ship jq, and scan-image classifies every VEX document with jq -e '...' >/dev/null 2>&1 — so a missing jq is indistinguishable from "not a VEX document" and all 1105 docs are discarded.

The timing confirms it: arm64 "skipped" 1105 documents in 830 ms (~0.75 ms each, too fast to have forked a process) against ~32 ms per document on amd64.

Fix is open as stackstate-ci-images#16 (STAC-25559). Once that image is built and the arm64-docker scale set picks up the new tag, these two jobs should go green with no change needed here.

LouisParkin and others added 3 commits August 10, 2026 10:20
…itHub

Closes the last two GitLab jobs with no GitHub counterpart: sign_deb and
pre_release_deb (.gitlab-ci-agent.yml lines 604 and 635).

Security prerequisite, not cleanup
----------------------------------
sign_debian_package.sh called printenv unconditionally. On GitLab that dumped
the GPG private key and its passphrase into the job log; stackstate-agent is a
PUBLIC repo, so on GitHub that log is world-readable. It also left the exported
private key in the checkout as gpg_private.key, where any later artifact upload
would collect it. Both are removed before any signing secret is wired in.

Key setup now happens in an ephemeral GNUPGHOME created with mktemp and mode
700, removed by an EXIT trap that also kills the gpg-agent so a preset
passphrase cannot outlive the job on a reused runner. The passphrase reaches
gpg through a mode-600 file inside that directory rather than argv.

Fixes a latent signing bug
--------------------------
The old preset step interpolated an unquoted command substitution into a single
gpg-preset-passphrase call. A key exposes one keygrip per primary and subkey, so
with a signing subkey the second keygrip was passed as a stray argument and
never presetted, leaving signing able to block on a pinentry prompt no CI runner
can answer. Each keygrip is now presetted individually. Verified against a
throwaway key: two keygrips, both presetted.

Shared setup
------------
sign_debian_package.sh and publish_package.sh need the same key in the same
state but are separate processes, and on GitHub may be separate steps, so
neither can rely on a keyring the other left behind. The setup moves into
gpg_signing_setup.sh, sourced by both. gpg-preset-passphrase is auto-detected
across /usr/lib/gnupg2, /usr/lib/gnupg, /usr/libexec and PATH, and the script
fails loudly rather than silently skipping the preset when it is absent.

Both scripts now fall back from CI_PROJECT_DIR to GITHUB_WORKSPACE, and
publish_package.sh falls back from CI_COMMIT_REF_NAME to GITHUB_REF_NAME, so the
apt codename stays the release branch exactly as it was on GitLab.

Workflow
--------
sign-and-publish-deb downloads both architecture artifacts and runs a single
deb-s3 upload. GitLab fanned this out per architecture, so two jobs rewrote the
same apt index concurrently; collecting both first removes that race.

The install script is split in two. generate-install-script runs
inv release.generate-install -t inside the build container, because the task
collection imports python-gitlab and the GitHub and Datadog API helpers and only
loads in the conda environment, and asserts the rendered script contains no
None.s3.amazonaws.com from an unset bucket variable. publish-install-script then
uploads it, so the container job never holds a credential.

Both publishing jobs are gated on push and bound to the agent-pre-release
environment, which carries the deployment branch rule, the signing secrets and
the AWS role from STAC-25545. Pull requests cannot reach them.

Validated: shellcheck -x clean on all three scripts; gpg_signing_setup exercised
against a generated throwaway key covering explicit override, PATH
auto-detection and the missing-binary failure path; actionlint clean apart from
the pre-existing self-hosted runner-label notices; zizmor reports no findings.

Blocked until the agent-pre-release environment, its four SIGNING_* secrets and
AGENT_PRERELEASE_ROLE_ARN exist.

Refs STAC-25546, STAC-25545

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion

Addresses review feedback on PR #455.

[P1] The signing job installed the publisher with `gem install deb-s3`,
resolving the latest code at run time into a job that then executes it with
the package signing key and the pre-release AWS credentials in scope. GitLab
never did this: `sign_deb` ran in a pinned image with deb-s3 already baked in,
so the runtime resolve was a regression introduced by the port.

deb-s3 and its full runtime dependency tree are now pinned in
`.github/deb-s3-gems.sha256` and installed by `.github/scripts/install-deb-s3.sh`,
which fetches each gem at its exact version and verifies it against the SHA256
RubyGems publishes for that release before anything is installed or executed.

The manifest covers nine gems. base64, bigdecimal and logger are deliberately
excluded: aws-sdk-core requires them at ">= 0" and they are Ruby default gems
supplied by the distribution's own ruby package, so pinning them would force a
native build for no supply-chain gain.

The script also links the canonical executable when RubyGems installs a
versioned binstub, since publish_package.sh invokes deb-s3 by bare name, and
ends with a `deb-s3 help` smoke check that activates the whole pinned set so a
missing or incompatible gem fails at install time rather than midway through
publishing.

[P2] publish-install-script depended only on generate-install-script, so it
could overwrite the public install.sh even when the build, signing or apt
upload had failed. GitLab's pre_release_deb required sign_deb. It now requires
both generate-install-script and sign-and-publish-deb, restoring that release
boundary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolving the rebase conflict against the rewritten base took this branch's
whole copy of build-deb.yml, which predated the exception work on STAC-25457.
That silently reverted two lines, so the gate ran here with no exceptions loaded
and reported all 24 findings as unmanaged.

Restores the quay LOCAL_IMAGE name the evaluator matches exceptions on, and the
exceptions-path input. The workflow diff against the base branch is now
additions only.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@LouisParkin
LouisParkin force-pushed the STAC-25457-publish-lane branch from 61e2bce to 19fa3b9 Compare August 10, 2026 08:22
@LouisParkin
LouisParkin force-pushed the STAC-25500-deb-prerelease branch from 1a2d643 to 185e763 Compare August 10, 2026 08:22
@LouisParkin
LouisParkin merged commit 65a9c2f into STAC-25457-publish-lane Aug 10, 2026
36 checks passed
@LouisParkin
LouisParkin deleted the STAC-25500-deb-prerelease branch August 10, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants