Skip to content

feat(rc-mirror)!: support local paths and parity controls#274

Draft
cxymds wants to merge 1 commit into
cxymds/issue-1372-transfer-controlsfrom
cxymds/issue-1373-mirror
Draft

feat(rc-mirror)!: support local paths and parity controls#274
cxymds wants to merge 1 commit into
cxymds/issue-1372-transfer-controlsfrom
cxymds/issue-1373-mirror

Conversation

@cxymds

@cxymds cxymds commented Jul 21, 2026

Copy link
Copy Markdown
Member

Refs rustfs/backlog#1373

Background and user impact

The legacy rc mirror implementation accepted only remote roots, used an unbounded command-specific scheduler, and copied remote objects through whole-object memory buffers. It lacked safe local backup/restore workflows, shared filtering and retry controls, and conditional destination mutation.

What changed

  • supports local-to-RustFS, RustFS-to-local, and RustFS-to-RustFS tree synchronization; local-to-local remains explicitly unsupported
  • maps normalized relative paths deterministically and consumes paginated remote listings
  • reuses the shared transfer plan/executor for filters, concurrency, aggregate rate limiting, retries, continue-on-error, dry-run, and summaries
  • stages uploads and downloads, validates source snapshots, cleans temporary files, and never starts removals after copy failure
  • rejects symlink traversal and normalization collisions
  • preserves content type and the existing JSON output shape
  • applies If-None-Match for new remote objects and If-Match for overwrites/removals, including multipart completion
  • classifies bare HTTP 404 HeadObject responses as not found by status instead of relying on SDK display text

Root cause of the RustFS mirror E2E failure

The first exact-head full integration run reached 60 passing tests and 7 failures. Three were the known SSE-S3 RustFS baseline. The four mirror write failures occurred before upload: mirror rechecks the destination with HeadObject, RustFS returned a bare HTTP 404 for a missing object, and the AWS SDK display string was only service error. The old classifier searched that string for NotFound or NoSuchKey and incorrectly returned a retryable network error. The new classifier reads the raw HTTP status and service error code, with a regression test for an empty-body 404.

RustFS compatibility

RustFS origin/main at bb7bba32376356fede5b1afdfdfce4c3a40df9dc evaluates If-Match and If-None-Match during CompleteMultipartUpload and carries reliant conditional-write E2E coverage.

BREAKING behavior and migration

This PR updates the protected mirror behavior contract and is marked BREAKING. Mirror no longer falls back to unconditional remote copy when source metadata lookup fails, and missing ETags are never treated as equality. Prefer --concurrency; --parallel remains a visible compatibility alias. Output schemas, config schema, and exit-code definitions are unchanged.

Dependency chain

  1. feat(cli)!: add runtime discovery and replication workflows #272 command stack
  2. feat(transfer): add multi-source planning and global controls #271 transfer controls
  3. this PR, based on feat(transfer): add multi-source planning and global controls #271

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace: 131 core, 162 S3, 392 CLI library, 399 CLI binary, and all default integration/contract/doc tests passed
  • mirror roadmap tests execute in both CLI targets
  • mirror planner tests: 5/5
  • integration feature binary compiles

The updated exact head is 66421c4 and is queued for CI plus manual full integration against RustFS latest.

@cxymds
cxymds changed the base branch from cxymds/issue-1372-transfer-controls to main July 21, 2026 09:03
@cxymds cxymds closed this Jul 21, 2026
@cxymds cxymds reopened this Jul 21, 2026
@cxymds
cxymds changed the base branch from main to cxymds/issue-1372-transfer-controls July 21, 2026 09:03
@cxymds
cxymds force-pushed the cxymds/issue-1373-mirror branch from 9d7766c to 3bc69b3 Compare July 21, 2026 09:41
@cxymds
cxymds changed the base branch from cxymds/issue-1372-transfer-controls to main July 21, 2026 09:42
@cxymds cxymds closed this Jul 21, 2026
@cxymds cxymds reopened this Jul 21, 2026
@cxymds
cxymds changed the base branch from main to cxymds/issue-1372-transfer-controls July 21, 2026 09:42
@cxymds
cxymds force-pushed the cxymds/issue-1373-mirror branch from 3bc69b3 to 81b4f39 Compare July 21, 2026 09:51
@cxymds
cxymds changed the base branch from cxymds/issue-1372-transfer-controls to main July 21, 2026 09:51
@cxymds cxymds closed this Jul 21, 2026
@cxymds cxymds reopened this Jul 21, 2026
@cxymds
cxymds changed the base branch from main to cxymds/issue-1372-transfer-controls July 21, 2026 09:51
@cxymds
cxymds force-pushed the cxymds/issue-1373-mirror branch from 81b4f39 to e246515 Compare July 21, 2026 10:45
@cxymds
cxymds changed the base branch from cxymds/issue-1372-transfer-controls to main July 21, 2026 10:45
@cxymds cxymds closed this Jul 21, 2026
@cxymds cxymds reopened this Jul 21, 2026
@cxymds
cxymds changed the base branch from main to cxymds/issue-1372-transfer-controls July 21, 2026 10:45
@cxymds
cxymds force-pushed the cxymds/issue-1373-mirror branch from e246515 to 66421c4 Compare July 21, 2026 11:21
@cxymds cxymds closed this Jul 21, 2026
@cxymds cxymds reopened this Jul 21, 2026
@cxymds
cxymds changed the base branch from cxymds/issue-1372-transfer-controls to main July 21, 2026 11:24
@cxymds cxymds closed this Jul 21, 2026
@cxymds cxymds reopened this Jul 21, 2026
@cxymds
cxymds changed the base branch from main to cxymds/issue-1372-transfer-controls July 21, 2026 11:24
@cxymds

cxymds commented Jul 21, 2026

Copy link
Copy Markdown
Member Author

Exact-head RustFS latest validation for 66421c4 is complete.

  • PR Integration workflow: passed.
  • Manual full integration: 64 passed, 3 failed.
  • Every mirror test now passes, including local-to-RustFS, RustFS-to-RustFS, content-type preservation, and parallel remove synchronization.
  • The only failures are the same three SSE-S3 baseline failures present on main: cp --enc-s3, mv --enc-s3, and pipe --enc-s3.

Run: https://github.com/rustfs/cli/actions/runs/29825904100

The previous exact head had 60 passed and four mirror failures, so the bare-404 HeadObject regression fix closes the complete mirror delta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant