# Fix Remote URI Corruption and Leading Slash Truncation in `storage.join()`

[aayuxsh326]

Fix Remote URI Corruption and Leading Slash Truncation in storage.join()

Background & Problem

In src/flyte/storage/_storage.py:

def join(*paths: str) -> str:
    """
    Join multiple paths together. This is a wrapper around os.path.join.
    # TODO replace with proper join with fsspec root etc

    Args:
        paths: Paths to be joined.
    """
    return str(os.path.join(*paths))

1. Windows Path Corruption for Cloud URIs: On Windows, os.path.join("s3://bucket", "prefix", "file.txt") uses \ as separator, outputting "s3://bucket\\prefix\\file.txt". Cloud storage services and object stores reject backslashes in keys.
2. Subpaths with Leading Slash Wipe Out Prefix: In Python, os.path.join treats any argument beginning with / (e.g. storage.join("s3://bucket/dir", "/subfile.txt")) as an absolute root path and discards all preceding components, returning "/subfile.txt".
3. Empty arguments: Calling storage.join() with empty args should return "" cleanly.

User Review Required

Note

For remote paths (checked via is_remote(paths[0])), components will be joined using forward slashes (/), stripping redundant slashes from child components so prefixes are never dropped. For local paths, standard os.path.join behavior is preserved.

Proposed Changes

Core Library

[MODIFY] src/flyte/storage/_storage.py

• Update join(*paths: str) -> str to check if paths[0] is a remote URI (using is_remote(paths[0])).
• If remote: join using forward slashes /, stripping leading and trailing / from inner segments, preserving the protocol scheme.
• If local: use os.path.join(*paths).

Tests

[MODIFY] tests/internal/storage/test_storage.py

• Add test_storage_join() test covering:
  • S3 / GCS / ABFS / Flyte remote URIs with multiple path components.
  • Remote URIs where subpaths contain leading slashes (/sub/file.txt).
  • Local relative and absolute paths.
  • Empty call storage.join().

Verification Plan

Automated Tests

• Run Python verification tests exercising storage.join across remote URIs, Windows path styles, and leading-slash subpaths.

[runllm]

The current implementation of storage.join() in src/flyte/storage/_storage.py uses os.path.join, which causes issues with remote URIs: it introduces backslashes on Windows and discards prefixes if any component starts with a leading slash. The proposed fix is to detect remote URIs using is_remote(paths[0]) and, if remote, join components with forward slashes, stripping leading/trailing slashes from all but the first segment, thus preserving the URI scheme and avoiding prefix loss. For local paths, the function should continue to use os.path.join. Empty arguments should return an empty string. This approach is consistent with the helper _join_remote_path already used in Flyte's codebase for memory stores, which preserves URI schemes and joins with POSIX separators, as shown in memory.py and tested in test_agent.py.

Would you like a step-by-step breakdown of how to implement and test this change?

Sources:

• Flyte SDK _join_remote_path helper
• Flyte SDK test coverage for remote path joining

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @Herald to ask follow-up questions. See runllm.com for more!

[aayuxsh326]

@Herald any Updates ??

[runllm]

There is no evidence in the provided data sources that this issue has been fixed in the main Flyte codebase. The current implementation of storage.join() still uses os.path.join, which is known to cause the remote URI corruption and leading slash truncation issues you described. The proposed fix—using a remote-aware join for remote URIs and preserving os.path.join for local paths—matches the approach of the internal _join_remote_path helper, but this has not yet been adopted in storage.join() as per the latest available sources (memory.py, test_agent.py, storage/_storage.py).

Would you like a detailed step-by-step guide on how to implement this fix and add the necessary tests?

Sources:

• Flyte SDK _join_remote_path helper
• Flyte SDK test coverage for remote path joining
• storage/_storage.py

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @Herald to ask follow-up questions. See runllm.com for more!