diff --git a/AGENTS.md b/AGENTS.md index 63edb6cf..38a67757 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -44,7 +44,7 @@ The CLI is a Click app that wraps the auto-generated `cloudsmith-api` Python SDK ### Push + metadata coupling -`cli/commands/push.py` is the most complex command. Every `push ` subcommand accepts `--metadata-*` flags resolved via `metadata_common`. Push validates metadata both locally and against the API **before** any file upload so malformed SBOM/BuildInfo payloads cannot leave orphan packages behind. Failure behavior is configurable with precedence: `--on-metadata-failure` flag > `$CLOUDSMITH_METADATA_FAILURE_MODE` env > `metadata_failure_mode` config key > `error` default. The kwarg names listed in `METADATA_KWARG_NAMES` and `METADATA_FAILURE_MODE_KWARG` must be popped off the kwargs before they reach the API client, which will reject unknown keys. +`cli/commands/push.py` is the most complex command. The per-format subcommands and their options are generated from the API's package-format model, so format-specific client-side behaviour is layered on top of that loop rather than branching inside it — see the `deb`-only `--dsc-file` registration at the end of `create_push_handlers()`, which pairs with `cli/dsc_parser.py` to derive `sources_file`/`changes_file` from a Debian `.dsc`. Every `push ` subcommand accepts `--metadata-*` flags resolved via `metadata_common`. Push validates metadata both locally and against the API **before** any file upload so malformed SBOM/BuildInfo payloads cannot leave orphan packages behind. Failure behavior is configurable with precedence: `--on-metadata-failure` flag > `$CLOUDSMITH_METADATA_FAILURE_MODE` env > `metadata_failure_mode` config key > `error` default. The kwarg names listed in `METADATA_KWARG_NAMES` and `METADATA_FAILURE_MODE_KWARG` must be popped off the kwargs before they reach the API client, which will reject unknown keys. ### Output formatting convention diff --git a/CHANGELOG.md b/CHANGELOG.md index 03f87991..eddc6bd9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. ## [Unreleased] -## [1.22.0] - 2026-08-11 - ### Added +- `cloudsmith push deb` now derives a Debian source package's members from its `.dsc`, so `cloudsmith push deb /// foo_1.0-1.dsc` is enough where `--sources-file` and `--changes-file` previously had to be worked out by hand (and their suffixes vary: `.orig.tar.gz`, `.orig.tar.bz2`, `.debian.tar.xz`, `.diff.gz`, ...). The `Checksums-Sha256:` or `Files:` field of the `.dsc` is read — plain or OpenPGP-clearsigned — and the upstream/native source archive becomes `--sources-file` while the Debian packaging archive becomes `--changes-file`, for the `1.0`, `2.0`, `3.0 (native)` and `3.0 (quilt)` source formats. `--dsc-file` names a `.dsc` other than `PACKAGE_FILE`, and an explicit `--sources-file` or `--changes-file` still wins for its own field. A detached upstream signature (`*.orig.tar.*.asc`) is skipped with a warning, since the deb package format has no field to carry it; a multi-component source package (`*.orig-.tar.*`) is rejected outright, because leaving a component behind would upload incomplete source. - `cloudsmith domains list` lists the hosts Cloudsmith can authenticate as a versioned JSON document: `{"version": 1, "domains": [{"host": ..., "format": ..., "type": ..., "domain_type": ..., "org": ..., "repository": ..., "primary": ..., "created_at": ...}]}`. The built-in list can be replaced by a `[domains]` section in a trusted `config.ini` — each entry maps a hostname to the format it serves, or to `download`/`upload` — for dedicated deployments. An organisation's own custom domains are listed ahead of the built-in hosts, and a custom domain that is disabled or not yet validated is left out entirely, since it serves nothing. `--format` and `--repo` narrow the list to the hosts usable for a package format or repository, most-preferred first, and `--domain-type` to those with one purpose: `download`, `upload`, `api` or `native_api`. - The Cloudsmith organisation is now named by `--org`, with `--organization` and `--oidc-org` accepted as aliases for the same option, and `org`, `organization` or `oidc_org` accepted in `config.ini`. `--oidc-org` named the setting after the first feature that wanted it; it is read by custom-domain discovery as well as OIDC token exchange, so it is now named after what it is. The `CLOUDSMITH_ORG` environment variable is unchanged, and `credential-helper install` no longer has a separate `--org` of its own. diff --git a/README.md b/README.md index 545edfb7..b6463c66 100644 --- a/README.md +++ b/README.md @@ -363,6 +363,12 @@ For example, if you wanted to upload a Debian package, you can do it in one-step cloudsmith push deb your-org/your-repo/ubuntu/xenial libxml2-2.9.4-2.x86_64.deb ``` +To upload a Debian *source* package, give it the `.dsc`. The source and Debian packaging archives listed in the `.dsc` are found and uploaded with it: + +``` +cloudsmith push deb your-org/your-repo/ubuntu/xenial hello_2.10-3.dsc +``` + Want to know how to do it with another packaging format? Easy, just ask for help: ``` diff --git a/cloudsmith_cli/cli/commands/push.py b/cloudsmith_cli/cli/commands/push.py index d16fa3e2..1dbe1b30 100644 --- a/cloudsmith_cli/cli/commands/push.py +++ b/cloudsmith_cli/cli/commands/push.py @@ -38,6 +38,7 @@ validate_create_package as api_validate_create_package, ) from .. import command, decorators, utils, validators +from ..dsc_parser import resolve_dsc_files from ..exceptions import handle_api_exceptions from ..metadata_common import ( MetadataContentError, @@ -75,6 +76,10 @@ #: separately from the metadata payload kwargs so it does not leak into the #: package-create API call. METADATA_FAILURE_MODE_KWARG = "cli_metadata_failure_mode" +#: Filename suffix of a Debian source control file. A ``deb`` push whose +#: PACKAGE_FILE carries it is a source-package upload, so its members can be +#: derived without ``--dsc-file`` (GitHub issue #56). +DSC_SUFFIX = ".dsc" def _metadata_failure_is_warn(opts=None): @@ -1019,6 +1024,13 @@ def upload_files_and_create_package( return slug_perm, slug +def _implied_dsc_file(package_file): + """Return ``package_file`` if it is itself a Debian ``.dsc``, else None.""" + if isinstance(package_file, str) and package_file.endswith(DSC_SUFFIX): + return package_file + return None + + def create_push_handlers(): """Create a handler for upload per package format.""" # pylint: disable=fixme @@ -1181,6 +1193,38 @@ def push_handler(ctx, *args, **kwargs): parameters = context.get(ctx.info_name) kwargs["package_type"] = ctx.info_name + # deb-only: derive the upstream/native source archive and Debian + # packaging archive from a .dsc (see GitHub issue #56). Only the + # `deb` subcommand ever registers --dsc-file (below), so this is + # a no-op kwargs.pop() for every other format. + dsc_file = kwargs.pop("dsc_file", None) + if dsc_file is None and not kwargs.get("sources_file"): + # Pushing a Debian source package means passing the .dsc as + # PACKAGE_FILE, and the API rejects it without a sources + # archive, so parse it by default rather than making the user + # name the same file twice. An explicit --sources-file means + # the caller is driving the members manually; leave them to it. + dsc_file = _implied_dsc_file(kwargs.get("package_file")) + if dsc_file: + resolved_dsc = resolve_dsc_files(dsc_file) + if resolved_dsc.ignored_files: + click.secho( + "Not uploading {files}: the deb package format has no " + "field for detached upstream signatures.".format( + files=", ".join(resolved_dsc.ignored_files) + ), + fg="yellow", + err=utils.should_use_stderr(opts), + ) + # Precedence: explicit --sources-file/--changes-file always + # win over values derived from --dsc-file. A user who passes + # both wants a manual override, not to have their explicit + # flag silently clobbered. + if not kwargs.get("sources_file"): + kwargs["sources_file"] = resolved_dsc.sources_file + if not kwargs.get("changes_file"): + kwargs["changes_file"] = resolved_dsc.changes_file + owner_repo = kwargs.pop("owner_repo") if "distribution" in parameters: kwargs["distribution"] = "/".join(owner_repo[2:]) @@ -1307,6 +1351,39 @@ def push_handler(ctx, *args, **kwargs): ) push_handler = decorator(push_handler) + if key == "deb": + # deb-only shim (GitHub issue #56): the generic per-format loop + # above is driven entirely by the API's package-format model, so + # this stays layered on top rather than becoming a new branch in + # that generic system. + push_handler = click.option( + "--dsc-file", + "dsc_file", + type=ExpandPath( + dir_okay=False, exists=True, writable=False, resolve_path=True + ), + default=None, + help=( + "Path to a Debian .dsc control file, plain or " + "OpenPGP-clearsigned. Only needed to point at a .dsc " + "other than PACKAGE_FILE: pushing a source package means " + "passing the .dsc as PACKAGE_FILE, and that is parsed " + "automatically unless --sources-file is given. Its " + "'Checksums-Sha256:' (or 'Files:') field supplies " + "--sources-file with the upstream/native source archive " + "and --changes-file with the Debian packaging archive " + "(.debian.tar.* or .diff.gz). Supports the 1.0, 2.0, " + "3.0 (native) and 3.0 (quilt) source formats, and the " + "referenced files must sit next to the .dsc. An explicit " + "--sources-file or --changes-file always takes precedence " + "over the derived value for that field. Detached upstream " + "signatures (*.asc) are skipped with a warning; a " + "multi-component source package (*.orig-*.tar.*) is " + "rejected, because the deb package format has no field " + "for the extra components." + ), + )(push_handler) + handlers[key] = push_handler diff --git a/cloudsmith_cli/cli/dsc_parser.py b/cloudsmith_cli/cli/dsc_parser.py new file mode 100644 index 00000000..0e8ee3e6 --- /dev/null +++ b/cloudsmith_cli/cli/dsc_parser.py @@ -0,0 +1,268 @@ +"""Parse Debian ``.dsc`` control files for ``push deb`` source uploads.""" + +import os +from dataclasses import dataclass, field +from email.parser import Parser + +import click + +_CLEAR_SIGNED_MESSAGE = "-----BEGIN PGP SIGNED MESSAGE-----" +_SIGNATURE = "-----BEGIN PGP SIGNATURE-----" +_DETACHED_SIGNATURE_SUFFIX = ".asc" +_FILE_LIST_FIELDS = ("Checksums-Sha256", "Files") +_QUILT_FORMATS = {"2.0", "3.0 (quilt)"} +_NATIVE_FORMAT = "3.0 (native)" + + +@dataclass(frozen=True) +class ResolvedDscFiles: + """Source-package members resolved from a Debian ``.dsc``.""" + + sources_file: str + changes_file: str | None = None + #: Members deliberately left out of the upload, for the caller to report. + ignored_files: tuple[str, ...] = field(default_factory=tuple) + + +def _usage_error(dsc_path, message): + # Names the file rather than the option: the .dsc is just as often taken + # from PACKAGE_FILE as from an explicit --dsc-file. + return click.UsageError(f"Debian source control file {dsc_path!r} {message}") + + +def _unwrap_clearsigned_control(text, dsc_path): + """Return deb822 control text from an optional OpenPGP cleartext signature.""" + lines = text.splitlines(keepends=True) + if not lines or lines[0].rstrip("\r\n") != _CLEAR_SIGNED_MESSAGE: + return text + + index = 1 + while index < len(lines) and lines[index].rstrip("\r\n"): + index += 1 + if index == len(lines): + raise _usage_error(dsc_path, "has a malformed OpenPGP cleartext signature.") + + cleartext = [] + for line in lines[index + 1 :]: + if line.rstrip("\r\n") == _SIGNATURE: + return "".join(cleartext) + # RFC 9580 cleartext signatures dash-escape lines beginning with "-". + cleartext.append(line.removeprefix("- ")) + + raise _usage_error(dsc_path, "has a malformed OpenPGP cleartext signature.") + + +def _read_control_message(dsc_path): + """Parse ``dsc_path`` as a plain or OpenPGP-clearsigned deb822 stanza.""" + try: + with open(dsc_path, encoding="utf-8", errors="replace") as dsc_fh: + text = dsc_fh.read() + except OSError as exc: + raise _usage_error(dsc_path, f"could not be read: {exc}") from exc + + return Parser().parsestr(_unwrap_clearsigned_control(text, dsc_path)) + + +def _parse_file_list(field_name, field_value, dsc_path): + filenames = [] + for line in field_value.splitlines(): + if not line.strip(): + continue + parts = line.split() + if len(parts) != 3: + raise _usage_error( + dsc_path, + f"has a malformed {field_name!r} entry {line.strip()!r}; " + "expected ' '.", + ) + filenames.append(parts[2]) + return filenames + + +def _extract_filenames(message, dsc_path): + """Return an agreed filename list, preferring the strong SHA-256 field.""" + file_lists = {} + for field_name in _FILE_LIST_FIELDS: + field_value = message.get(field_name) + if field_value: + filenames = _parse_file_list(field_name, field_value, dsc_path) + if filenames: + file_lists[field_name] = filenames + + if not file_lists: + raise _usage_error( + dsc_path, + "has no non-empty 'Checksums-Sha256:' or 'Files:' field to parse.", + ) + + for filenames in file_lists.values(): + if len(filenames) != len(set(filenames)): + raise _usage_error(dsc_path, "lists the same source-package member twice.") + + selected_name = next(iter(file_lists)) + selected = file_lists[selected_name] + for field_name, filenames in file_lists.items(): + if set(filenames) != set(selected): + raise _usage_error( + dsc_path, + f"has conflicting filenames in {selected_name!r} and {field_name!r}.", + ) + + return selected + + +def _required_field(message, field_name, dsc_path): + value = message.get(field_name) + if not value or not value.strip(): + raise _usage_error(dsc_path, f"has no {field_name!r} field.") + return value.strip() + + +def _validate_member_names(filenames, dsc_path): + for filename in filenames: + if os.path.isabs(filename) or os.path.basename(filename) != filename: + raise _usage_error( + dsc_path, + f"references invalid member filename {filename!r}; source-package " + "members must be filenames next to the .dsc, not paths.", + ) + + +def _is_tar_archive(filename, stem): + prefix = f"{stem}.tar." + return filename.startswith(prefix) and len(filename) > len(prefix) + + +def _classify_members(message, filenames, dsc_path): + """Map source-package members to the two fields accepted by the SDK.""" + source = _required_field(message, "Source", dsc_path) + version = _required_field(message, "Version", dsc_path).split(":", 1)[-1] + source_format = _required_field(message, "Format", dsc_path) + upstream_version = version.rsplit("-", 1)[0] + + # Detached upstream signatures (e.g. hello_2.10.orig.tar.gz.asc) are + # common and have no field in the deb upload model. Nothing in the + # uploaded source is lost by leaving them out, so they are skipped with a + # warning rather than failing the push. + signature_files = [ + filename + for filename in filenames + if filename.endswith(_DETACHED_SIGNATURE_SUFFIX) + ] + # Classify the uploadable members only. A signature shares its tarball's + # stem (`*.orig.tar.gz.asc`), so leaving it in would match as a second + # source archive. + members = [f for f in filenames if f not in set(signature_files)] + + sources = [] + changes = [] + component_files = [] + legacy_non_native = False + + if source_format in _QUILT_FORMATS: + orig_stem = f"{source}_{upstream_version}.orig" + component_prefix = f"{orig_stem}-" + debian_stem = f"{source}_{version}.debian" + sources = [f for f in members if _is_tar_archive(f, orig_stem)] + changes = [f for f in members if _is_tar_archive(f, debian_stem)] + component_files = [ + f + for f in members + if f.startswith(component_prefix) and ".tar." in f[len(component_prefix) :] + ] + elif source_format == "1.0": + orig_stem = f"{source}_{upstream_version}.orig" + diff_name = f"{source}_{version}.diff.gz" + native_stem = f"{source}_{version}" + orig_files = [f for f in members if _is_tar_archive(f, orig_stem)] + diff_files = [f for f in members if f == diff_name] + native_files = [f for f in members if _is_tar_archive(f, native_stem)] + if orig_files or diff_files: + legacy_non_native = True + sources = orig_files + changes = diff_files + else: + sources = native_files + elif source_format == _NATIVE_FORMAT: + sources = [f for f in members if _is_tar_archive(f, f"{source}_{version}")] + else: + # '3.0 (git)' ships a git bundle and '3.0 (bzr)' a VCS tarball, neither + # of which is a source archive the deb upload model can index, so both + # fall through to the unsupported-format error alongside + # '3.0 (custom)'. + raise _usage_error( + dsc_path, f"uses unsupported Debian source format {source_format!r}." + ) + + if component_files: + raise _usage_error( + dsc_path, + "references a multi-component source package ({files}). Cloudsmith " + "does not support multi-component Debian source packages for deb " + "uploads.".format(files=", ".join(component_files)), + ) + + classified = set(sources + changes + component_files) + unexpected = [f for f in members if f not in classified] + if unexpected: + raise _usage_error( + dsc_path, + "contains unsupported or incorrectly named source-package member(s) " + f"({', '.join(unexpected)}) for format {source_format!r}.", + ) + if len(sources) != 1: + raise _usage_error( + dsc_path, + f"must reference exactly one main source archive for format " + f"{source_format!r}; found {len(sources)}.", + ) + if source_format in _QUILT_FORMATS and len(changes) != 1: + raise _usage_error( + dsc_path, + f"must reference exactly one Debian packaging archive for format " + f"{source_format!r}; found {len(changes)}.", + ) + if legacy_non_native and len(changes) != 1: + raise _usage_error( + dsc_path, + "must reference exactly one Debian packaging diff for non-native " + f"format '1.0'; found {len(changes)}.", + ) + + return sources[0], changes[0] if changes else None, tuple(signature_files) + + +def _resolve_member(base_dir, filename, dsc_path): + """Canonicalise a member that ``_validate_member_names`` has vetted.""" + candidate = os.path.join(base_dir, filename) + if not os.path.isfile(candidate): + raise _usage_error( + dsc_path, + f"references {filename!r}, but it is not a regular file next to the .dsc.", + ) + # Uploading the canonical path means a symlink swapped between this check + # and the upload cannot redirect the read. The link itself may point + # outside the directory: `mk-origtargz --symlink` (the uscan default) + # routinely symlinks the .orig tarball in from a download cache. + return os.path.realpath(candidate) + + +def resolve_dsc_files(dsc_path): + """Return the :class:`ResolvedDscFiles` derived from a Debian ``.dsc``.""" + message = _read_control_message(dsc_path) + filenames = _extract_filenames(message, dsc_path) + _validate_member_names(filenames, dsc_path) + source_filename, changes_filename, ignored_files = _classify_members( + message, filenames, dsc_path + ) + + base_dir = os.path.realpath(os.path.dirname(os.path.abspath(dsc_path))) + return ResolvedDscFiles( + sources_file=_resolve_member(base_dir, source_filename, dsc_path), + changes_file=( + _resolve_member(base_dir, changes_filename, dsc_path) + if changes_filename + else None + ), + ignored_files=ignored_files, + ) diff --git a/cloudsmith_cli/cli/tests/test_dsc_parser.py b/cloudsmith_cli/cli/tests/test_dsc_parser.py new file mode 100644 index 00000000..3722d7d8 --- /dev/null +++ b/cloudsmith_cli/cli/tests/test_dsc_parser.py @@ -0,0 +1,271 @@ +"""Tests for Debian ``.dsc`` parsing and member resolution.""" + +import os + +import click +import pytest + +from ..dsc_parser import ResolvedDscFiles, resolve_dsc_files + + +def _file_field(name, filenames): + checksum = "a" * (64 if name == "Checksums-Sha256" else 32) + entries = "\n".join(f" {checksum} 100 {filename}" for filename in filenames) + return f"{name}:\n{entries}\n" + + +def _write_dsc( + tmp_path, + filenames, + *, + source="pkg", + version="1.0", + source_format="3.0 (native)", + fields=("Files",), + clearsigned=False, + name=None, +): + control = ( + f"Format: {source_format}\nSource: {source}\nVersion: {version}\n" + + "".join(_file_field(field, filenames) for field in fields) + ) + if clearsigned: + control = ( + "-----BEGIN PGP SIGNED MESSAGE-----\n" + "Hash: SHA256\n\n" + f"{control}" + "-----BEGIN PGP SIGNATURE-----\n" + "test-signature-data\n" + "-----END PGP SIGNATURE-----\n" + ) + dsc_path = tmp_path / (name or f"{source}_{version}.dsc") + dsc_path.write_text(control) + return dsc_path + + +def _touch(tmp_path, name): + path = tmp_path / name + path.write_bytes(b"dummy content") + return path + + +def test_resolves_native_source_archive(tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.tar.xz") + dsc_path = _write_dsc(tmp_path, [source_archive.name]) + + assert resolve_dsc_files(str(dsc_path)) == ResolvedDscFiles(str(source_archive)) + + +def test_resolves_quilt_source_and_debian_archives(tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.orig.tar.gz") + debian_archive = _touch(tmp_path, "pkg_1.0-1.debian.tar.xz") + dsc_path = _write_dsc( + tmp_path, + [source_archive.name, debian_archive.name], + version="1.0-1", + source_format="3.0 (quilt)", + ) + + assert resolve_dsc_files(str(dsc_path)) == ResolvedDscFiles( + str(source_archive), str(debian_archive) + ) + + +def test_resolves_legacy_non_native_source_and_diff_archives(tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.orig.tar.gz") + diff_archive = _touch(tmp_path, "pkg_1.0-1.diff.gz") + dsc_path = _write_dsc( + tmp_path, + [source_archive.name, diff_archive.name], + version="1.0-1", + source_format="1.0", + ) + + assert resolve_dsc_files(str(dsc_path)) == ResolvedDscFiles( + str(source_archive), str(diff_archive) + ) + + +def test_rejects_legacy_non_native_source_without_diff(tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.orig.tar.gz") + dsc_path = _write_dsc( + tmp_path, + [source_archive.name], + version="1.0-1", + source_format="1.0", + ) + + with pytest.raises(click.UsageError, match="exactly one Debian packaging diff"): + resolve_dsc_files(str(dsc_path)) + + +def test_resolves_clearsigned_dsc_using_matching_sha256_and_files(tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.orig.tar.gz") + debian_archive = _touch(tmp_path, "pkg_1.0-1.debian.tar.xz") + dsc_path = _write_dsc( + tmp_path, + [source_archive.name, debian_archive.name], + version="1.0-1", + source_format="3.0 (quilt)", + fields=("Files", "Checksums-Sha256"), + clearsigned=True, + ) + + assert resolve_dsc_files(str(dsc_path)) == ResolvedDscFiles( + str(source_archive), str(debian_archive) + ) + + +def test_uses_checksums_sha256_when_files_is_absent(tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.tar.gz") + dsc_path = _write_dsc(tmp_path, [source_archive.name], fields=("Checksums-Sha256",)) + + assert resolve_dsc_files(str(dsc_path)) == ResolvedDscFiles(str(source_archive)) + + +def test_rejects_conflicting_checksum_file_lists(tmp_path): + dsc_path = tmp_path / "pkg_1.0.dsc" + dsc_path.write_text( + "Format: 3.0 (native)\nSource: pkg\nVersion: 1.0\n" + + _file_field("Files", ["pkg_1.0.tar.gz"]) + + _file_field("Checksums-Sha256", ["other_1.0.tar.gz"]) + ) + + with pytest.raises(click.UsageError, match="conflicting filenames"): + resolve_dsc_files(str(dsc_path)) + + +def test_rejects_malformed_file_list_row(tmp_path): + dsc_path = tmp_path / "pkg_1.0.dsc" + dsc_path.write_text( + "Format: 3.0 (native)\nSource: pkg\nVersion: 1.0\n" + "Files:\n deadbeef pkg_1.0.tar.gz\n" + ) + + with pytest.raises(click.UsageError, match="malformed 'Files' entry"): + resolve_dsc_files(str(dsc_path)) + + +def test_rejects_missing_referenced_file(tmp_path): + dsc_path = _write_dsc(tmp_path, ["pkg_1.0.tar.gz"]) + + with pytest.raises(click.UsageError, match="not a regular file next to"): + resolve_dsc_files(str(dsc_path)) + + +def test_rejects_multi_component_source_package(tmp_path): + _touch(tmp_path, "pkg_1.0.orig.tar.gz") + _touch(tmp_path, "pkg_1.0.orig-libbar.tar.gz") + _touch(tmp_path, "pkg_1.0-1.debian.tar.xz") + dsc_path = _write_dsc( + tmp_path, + [ + "pkg_1.0.orig.tar.gz", + "pkg_1.0.orig-libbar.tar.gz", + "pkg_1.0-1.debian.tar.xz", + ], + version="1.0-1", + source_format="3.0 (quilt)", + ) + + with pytest.raises(click.UsageError, match="multi-component"): + resolve_dsc_files(str(dsc_path)) + + +def test_skips_detached_signature_without_failing(tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.orig.tar.gz") + _touch(tmp_path, "pkg_1.0.orig.tar.gz.asc") + debian_archive = _touch(tmp_path, "pkg_1.0-1.debian.tar.xz") + dsc_path = _write_dsc( + tmp_path, + [ + "pkg_1.0.orig.tar.gz", + "pkg_1.0.orig.tar.gz.asc", + "pkg_1.0-1.debian.tar.xz", + ], + version="1.0-1", + source_format="3.0 (quilt)", + ) + + assert resolve_dsc_files(str(dsc_path)) == ResolvedDscFiles( + str(source_archive), + str(debian_archive), + ignored_files=("pkg_1.0.orig.tar.gz.asc",), + ) + + +@pytest.mark.parametrize("source_format", ["3.0 (git)", "3.0 (bzr)", "3.0 (custom)"]) +def test_rejects_source_formats_without_an_indexable_archive(source_format, tmp_path): + _touch(tmp_path, "pkg_1.0.git") + dsc_path = _write_dsc(tmp_path, ["pkg_1.0.git"], source_format=source_format) + + with pytest.raises(click.UsageError, match="unsupported Debian source format"): + resolve_dsc_files(str(dsc_path)) + + +@pytest.mark.parametrize("filename", ["/etc/passwd", "../pkg_1.0.tar.gz"]) +def test_rejects_member_paths(filename, tmp_path): + dsc_path = _write_dsc(tmp_path, [filename]) + + with pytest.raises(click.UsageError, match="must be filenames next to the .dsc"): + resolve_dsc_files(str(dsc_path)) + + +def test_resolves_symlinked_member_to_its_canonical_target(tmp_path): + # `mk-origtargz --symlink` (the uscan default) links the .orig tarball in + # from a download cache, so a member pointing outside the .dsc directory + # is a normal build tree, not an attempt to escape it. + dsc_dir = tmp_path / "source-package" + dsc_dir.mkdir() + outside_archive = _touch(tmp_path, "outside.tar.gz") + (dsc_dir / "pkg_1.0.tar.gz").symlink_to(outside_archive) + dsc_path = _write_dsc(dsc_dir, ["pkg_1.0.tar.gz"]) + + assert resolve_dsc_files(str(dsc_path)) == ResolvedDscFiles(str(outside_archive)) + + +def test_rejects_member_symlinked_to_a_directory(tmp_path): + dsc_dir = tmp_path / "source-package" + dsc_dir.mkdir() + (dsc_dir / "pkg_1.0.tar.gz").symlink_to(tmp_path, target_is_directory=True) + dsc_path = _write_dsc(dsc_dir, ["pkg_1.0.tar.gz"]) + + with pytest.raises(click.UsageError, match="not a regular file next to"): + resolve_dsc_files(str(dsc_path)) + + +@pytest.mark.parametrize( + ("source", "version", "filename"), + [ + ("foo.orig-bar", "1.0", "foo.orig-bar_1.0.tar.gz"), + ("foo", "1.0.orig-bar", "foo_1.0.orig-bar.tar.gz"), + ], +) +def test_native_name_or_version_containing_orig_marker_is_not_a_component( + source, version, filename, tmp_path +): + source_archive = _touch(tmp_path, filename) + dsc_path = _write_dsc( + tmp_path, [filename], source=source, version=version, name="package.dsc" + ) + + assert resolve_dsc_files(str(dsc_path)) == ResolvedDscFiles(str(source_archive)) + + +def test_rejects_missing_file_listing(tmp_path): + dsc_path = tmp_path / "empty.dsc" + dsc_path.write_text("Format: 3.0 (native)\nSource: pkg\nVersion: 1.0\n") + + with pytest.raises(click.UsageError, match="Checksums-Sha256"): + resolve_dsc_files(str(dsc_path)) + + +def test_resolved_paths_are_canonical(tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.tar.gz") + internal_link = tmp_path / "linked" + internal_link.symlink_to(tmp_path, target_is_directory=True) + dsc_path = _write_dsc(tmp_path, [source_archive.name]) + + resolved = resolve_dsc_files(os.path.join(str(internal_link), dsc_path.name)) + + assert resolved.sources_file == str(source_archive) diff --git a/cloudsmith_cli/cli/tests/test_push_dsc_cli.py b/cloudsmith_cli/cli/tests/test_push_dsc_cli.py new file mode 100644 index 00000000..12f2c544 --- /dev/null +++ b/cloudsmith_cli/cli/tests/test_push_dsc_cli.py @@ -0,0 +1,374 @@ +"""Registered-command tests for ``cloudsmith push deb --dsc-file``.""" + +from pathlib import Path +from unittest.mock import patch + +import pytest + +from .. import config as cli_config +from ..commands.main import main + +HERMETIC_ARGS = ["--api-key", "fake-api-key"] + +_MOCK_TARGETS = ( + "cloudsmith_cli.cli.commands.push.validate_create_package", + "cloudsmith_cli.cli.commands.push.validate_upload_file", + "cloudsmith_cli.cli.commands.push.upload_file", + "cloudsmith_cli.cli.commands.push.create_package", + "cloudsmith_cli.cli.commands.push.wait_for_package_sync", +) + + +@pytest.fixture(autouse=True) +def hermetic_environment(monkeypatch): + """Keep developer environment and config values out of command tests.""" + monkeypatch.delenv("CLOUDSMITH_ORG", raising=False) + monkeypatch.delenv("CLOUDSMITH_CONFIG_FILE", raising=False) + monkeypatch.delattr(cli_config.OPTIONS, "value", raising=False) + monkeypatch.setattr(cli_config.ConfigReader, "config_files", ["config.ini"]) + monkeypatch.setattr(cli_config.ConfigReader, "config_searchpath", ["."]) + + +def _touch(tmp_path, name): + path = tmp_path / name + path.write_bytes(b"dummy content") + return path + + +def _file_field(name, filenames): + checksum = "a" * (64 if name == "Checksums-Sha256" else 32) + entries = "\n".join(f" {checksum} 100 {filename}" for filename in filenames) + return f"{name}:\n{entries}\n" + + +def _write_dsc( + tmp_path, + filenames, + *, + source="pkg", + version="1.0", + source_format="3.0 (native)", + clearsigned=False, +): + control = ( + f"Format: {source_format}\nSource: {source}\nVersion: {version}\n" + + _file_field("Files", filenames) + + _file_field("Checksums-Sha256", filenames) + ) + if clearsigned: + control = ( + "-----BEGIN PGP SIGNED MESSAGE-----\n" + "Hash: SHA256\n\n" + f"{control}" + "-----BEGIN PGP SIGNATURE-----\n" + "test-signature-data\n" + "-----END PGP SIGNATURE-----\n" + ) + dsc_path = tmp_path / f"{source}_{version}.dsc" + dsc_path.write_text(control) + return dsc_path + + +def _upload_identifier(*, filepath, **_kwargs): + return f"uploaded:{Path(filepath).name}" + + +def _invoke(runner, tmp_path, extra_args, mocks, package_file=None): + if package_file is None: + package_file = _touch(tmp_path, "pkg_1.0-1_amd64.deb") + with ( + patch(_MOCK_TARGETS[0]) as mock_validate_create_package, + patch(_MOCK_TARGETS[1], return_value="checksum") as mock_validate_upload, + patch(_MOCK_TARGETS[2], side_effect=_upload_identifier) as mock_upload, + patch( + _MOCK_TARGETS[3], return_value=("slug-perm", "slug") + ) as mock_create_package, + patch(_MOCK_TARGETS[4]), + ): + mocks.update( + validate_create_package=mock_validate_create_package, + validate_upload_file=mock_validate_upload, + upload_file=mock_upload, + create_package=mock_create_package, + ) + result = runner.invoke( + main, + [ + "push", + "deb", + "example/repo/ubuntu/xenial", + str(package_file), + *extra_args, + *HERMETIC_ARGS, + ], + catch_exceptions=False, + ) + return result + + +@pytest.mark.parametrize( + ("source_format", "version", "source_name", "changes_name"), + [ + ("3.0 (native)", "1.0", "pkg_1.0.tar.xz", None), + ( + "3.0 (quilt)", + "1.0-1", + "pkg_1.0.orig.tar.gz", + "pkg_1.0-1.debian.tar.xz", + ), + ("1.0", "1.0-1", "pkg_1.0.orig.tar.gz", "pkg_1.0-1.diff.gz"), + ], +) +def test_dsc_maps_real_source_package_members_to_uploaded_sdk_fields( + runner, tmp_path, source_format, version, source_name, changes_name +): + source_archive = _touch(tmp_path, source_name) + filenames = [source_archive.name] + if changes_name: + filenames.append(_touch(tmp_path, changes_name).name) + dsc_path = _write_dsc( + tmp_path, filenames, version=version, source_format=source_format + ) + + mocks = {} + result = _invoke(runner, tmp_path, ["--dsc-file", str(dsc_path)], mocks) + + assert result.exit_code == 0, result.output + validation_kwargs = mocks["validate_create_package"].call_args.kwargs + assert validation_kwargs["sources_file"] == str(source_archive) + assert validation_kwargs["changes_file"] == ( + str(tmp_path / changes_name) if changes_name else None + ) + create_kwargs = mocks["create_package"].call_args.kwargs + assert create_kwargs["sources_file"] == f"uploaded:{source_name}" + assert create_kwargs["changes_file"] == ( + f"uploaded:{changes_name}" if changes_name else None + ) + + +def test_dsc_as_package_file_is_parsed_without_the_option(runner, tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.orig.tar.gz") + debian_archive = _touch(tmp_path, "pkg_1.0-1.debian.tar.xz") + dsc_path = _write_dsc( + tmp_path, + [source_archive.name, debian_archive.name], + version="1.0-1", + source_format="3.0 (quilt)", + ) + + mocks = {} + result = _invoke(runner, tmp_path, [], mocks, package_file=dsc_path) + + assert result.exit_code == 0, result.output + kwargs = mocks["validate_create_package"].call_args.kwargs + assert kwargs["sources_file"] == str(source_archive) + assert kwargs["changes_file"] == str(debian_archive) + + +def test_explicit_sources_file_stops_the_package_file_being_parsed(runner, tmp_path): + # The .dsc references a member that is absent, so parsing it at all would + # abort the push; --sources-file means the caller drives the members. + explicit_source = _touch(tmp_path, "explicit.tar.gz") + dsc_path = _write_dsc(tmp_path, ["absent_1.0.tar.gz"]) + + mocks = {} + result = _invoke( + runner, + tmp_path, + ["--sources-file", str(explicit_source)], + mocks, + package_file=dsc_path, + ) + + assert result.exit_code == 0, result.output + kwargs = mocks["validate_create_package"].call_args.kwargs + assert kwargs["sources_file"] == str(explicit_source) + assert kwargs["changes_file"] is None + + +def test_binary_package_file_is_never_parsed_as_a_dsc(runner, tmp_path): + mocks = {} + result = _invoke(runner, tmp_path, [], mocks) + + assert result.exit_code == 0, result.output + kwargs = mocks["validate_create_package"].call_args.kwargs + assert kwargs["sources_file"] is None + assert kwargs["changes_file"] is None + + +def test_clearsigned_dsc_is_parsed_through_registered_command(runner, tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.orig.tar.gz") + debian_archive = _touch(tmp_path, "pkg_1.0-1.debian.tar.xz") + dsc_path = _write_dsc( + tmp_path, + [source_archive.name, debian_archive.name], + version="1.0-1", + source_format="3.0 (quilt)", + clearsigned=True, + ) + + mocks = {} + result = _invoke(runner, tmp_path, ["--dsc-file", str(dsc_path)], mocks) + + assert result.exit_code == 0, result.output + kwargs = mocks["validate_create_package"].call_args.kwargs + assert kwargs["sources_file"] == str(source_archive) + assert kwargs["changes_file"] == str(debian_archive) + + +def test_explicit_sources_file_wins_per_field(runner, tmp_path): + dsc_source = _touch(tmp_path, "pkg_1.0.orig.tar.gz") + debian_archive = _touch(tmp_path, "pkg_1.0-1.debian.tar.xz") + explicit_source = _touch(tmp_path, "explicit.tar.gz") + dsc_path = _write_dsc( + tmp_path, + [dsc_source.name, debian_archive.name], + version="1.0-1", + source_format="3.0 (quilt)", + ) + + mocks = {} + result = _invoke( + runner, + tmp_path, + [ + "--dsc-file", + str(dsc_path), + "--sources-file", + str(explicit_source), + ], + mocks, + ) + + assert result.exit_code == 0, result.output + kwargs = mocks["validate_create_package"].call_args.kwargs + assert kwargs["sources_file"] == str(explicit_source) + assert kwargs["changes_file"] == str(debian_archive) + + +def test_explicit_changes_file_wins_per_field(runner, tmp_path): + source_archive = _touch(tmp_path, "pkg_1.0.orig.tar.gz") + dsc_changes = _touch(tmp_path, "pkg_1.0-1.debian.tar.xz") + explicit_changes = _touch(tmp_path, "explicit.diff.gz") + dsc_path = _write_dsc( + tmp_path, + [source_archive.name, dsc_changes.name], + version="1.0-1", + source_format="3.0 (quilt)", + ) + + mocks = {} + result = _invoke( + runner, + tmp_path, + [ + "--dsc-file", + str(dsc_path), + "--changes-file", + str(explicit_changes), + ], + mocks, + ) + + assert result.exit_code == 0, result.output + kwargs = mocks["validate_create_package"].call_args.kwargs + assert kwargs["sources_file"] == str(source_archive) + assert kwargs["changes_file"] == str(explicit_changes) + + +@pytest.mark.parametrize("member", ["/etc/passwd", "../pkg_1.0.tar.gz"]) +def test_member_path_is_rejected_before_network_calls(runner, tmp_path, member): + dsc_path = _write_dsc(tmp_path, [member]) + + mocks = {} + result = _invoke(runner, tmp_path, ["--dsc-file", str(dsc_path)], mocks) + + assert result.exit_code != 0 + assert "must be filenames next to the .dsc" in result.output + mocks["validate_create_package"].assert_not_called() + mocks["validate_upload_file"].assert_not_called() + + +def test_symlinked_member_uploads_its_canonical_target(runner, tmp_path): + dsc_dir = tmp_path / "dsc" + dsc_dir.mkdir() + outside_archive = _touch(tmp_path, "outside.tar.gz") + (dsc_dir / "pkg_1.0.tar.gz").symlink_to(outside_archive) + dsc_path = _write_dsc(dsc_dir, ["pkg_1.0.tar.gz"]) + + mocks = {} + result = _invoke(runner, tmp_path, ["--dsc-file", str(dsc_path)], mocks) + + assert result.exit_code == 0, result.output + _, kwargs = mocks["create_package"].call_args + assert kwargs["sources_file"] == "uploaded:outside.tar.gz" + + +def test_member_symlinked_to_a_directory_is_rejected_before_network_calls( + runner, tmp_path +): + dsc_dir = tmp_path / "dsc" + dsc_dir.mkdir() + (dsc_dir / "pkg_1.0.tar.gz").symlink_to(tmp_path, target_is_directory=True) + dsc_path = _write_dsc(dsc_dir, ["pkg_1.0.tar.gz"]) + + mocks = {} + result = _invoke(runner, tmp_path, ["--dsc-file", str(dsc_path)], mocks) + + assert result.exit_code != 0 + assert "not a regular file next to the .dsc" in result.output + mocks["validate_create_package"].assert_not_called() + mocks["validate_upload_file"].assert_not_called() + + +def test_multi_component_is_rejected_before_network_calls(runner, tmp_path): + filenames = [ + _touch(tmp_path, "pkg_1.0.orig.tar.gz").name, + _touch(tmp_path, "pkg_1.0.orig-libbar.tar.gz").name, + _touch(tmp_path, "pkg_1.0-1.debian.tar.xz").name, + ] + dsc_path = _write_dsc( + tmp_path, filenames, version="1.0-1", source_format="3.0 (quilt)" + ) + + mocks = {} + result = _invoke(runner, tmp_path, ["--dsc-file", str(dsc_path)], mocks) + + assert result.exit_code != 0 + assert "multi-component" in result.output + mocks["validate_create_package"].assert_not_called() + + +def test_detached_signature_is_skipped_with_a_warning(runner, tmp_path): + filenames = [ + _touch(tmp_path, "pkg_1.0.orig.tar.gz").name, + _touch(tmp_path, "pkg_1.0.orig.tar.gz.asc").name, + _touch(tmp_path, "pkg_1.0-1.debian.tar.xz").name, + ] + dsc_path = _write_dsc( + tmp_path, filenames, version="1.0-1", source_format="3.0 (quilt)" + ) + + mocks = {} + result = _invoke(runner, tmp_path, ["--dsc-file", str(dsc_path)], mocks) + + assert result.exit_code == 0, result.output + assert "Not uploading pkg_1.0.orig.tar.gz.asc" in result.output + _, kwargs = mocks["create_package"].call_args + assert kwargs["sources_file"] == "uploaded:pkg_1.0.orig.tar.gz" + assert kwargs["changes_file"] == "uploaded:pkg_1.0-1.debian.tar.xz" + + +def test_push_deb_help_documents_dsc_file_option(runner): + result = runner.invoke(main, ["push", "deb", "--help"], catch_exceptions=False) + + assert result.exit_code == 0, result.output + assert "--dsc-file" in result.output + assert ".debian.tar.*" in result.output + + +def test_push_non_deb_format_has_no_dsc_file_option(runner): + result = runner.invoke(main, ["push", "raw", "--help"], catch_exceptions=False) + + assert result.exit_code == 0, result.output + assert "--dsc-file" not in result.output