README.md

SILO client

The client turns a versioned S3 bucket into a branchable file repository. Files remain ordinary whole S3 objects under immutable derived keys; Prolly trees provide snapshots and history.

Requirements

• an S3 or S3-compatible bucket with versioning enabled;
• conditional writes, exact-version reads, range reads, and strong read-after-write;
• a repository prefix reserved exclusively for this client;
• a stable writer identity and a trusted provider-attestation signer.

Do not write under the repository prefix outside this library.

Create a client

This complete program shape works with AWS SDK configuration you supply:

use std::{sync::Arc, time::Duration};
use silo_s3_client::{
    core::ProviderPerKeyVersionLimit,
    Client, HmacAttestationSigner, ProviderIdentity,
};

async fn create(
    aws: aws_sdk_s3::Client,
    bucket: &str,
) -> Result<Client, silo_s3_client::Error> {
    Client::builder()
        .aws_client(aws)
        .bucket(bucket)
        .repository_prefix(".prolly")
        .default_branch("main")
        .writer("ingest-worker-01")
        .provider_identity(ProviderIdentity::aws_region("us-west-2"))
        .provider_attestation_validity(Duration::from_secs(24 * 60 * 60))
        .attestation_signer(Arc::new(HmacAttestationSigner::single(
            "provider-key-2026-01",
            vec![0x41; 32],
        )?))
        .provider_per_key_version_limit(
            ProviderPerKeyVersionLimit::Finite(10_000),
        )
        .initialize()
        .await
}

Call initialize once for a new prefix. Use the same builder inputs with open after restart. The default prefix is .prolly.

Runnable scenario examples

The example suite uses an isolated repository prefix for every run. Start the pinned local RustFS service, then run every scenario:

docker compose -f docker-compose.rustfs.yml up -d
scripts/run_rustfs_examples.sh

Run one scenario with Cargo:

cargo run --locked --manifest-path Cargo.toml \
  -p silo-s3-client --example branch_diff_merge

Run the staged tiny-file performance benchmark (10K through 1M files by default). The repository grows cumulatively and reports write, publication, full-list, and sampled-read latency and throughput after every stage:

cargo run --release --locked --manifest-path Cargo.toml \
  -p silo-s3-client --example rustfs_small_files_benchmark

Use SILO_RUSTFS_PERF_STAGES=10000,20000 for a shorter run. RustFS defaults to the silo bucket and the repository's standard local credentials (siloadmin / silo-local-secret-change-me); all values remain overridable with the SILO_RUSTFS_* environment variables.

Example	Demonstrates
basic_object_workflow.rs	Metadata, ranges, copy, listing, and historical reads
atomic_batch_and_streaming.rs	Durable atomic batches, checkpoints, and streamed input
branch_diff_merge.rs	Branch isolation, bounded diff, structural merge, log, and reflog
restore_and_recovery.rs	Restartable restore, administrative reset, and reflog recovery
history_transfer_and_backup.rs	Commit-DAG transfer and logical backup verification
integrity_gc_and_observability.rs	Deep fsck, cache prewarm, metrics, retention pins, and GC

The default credentials and attestation key are local-demo values. Never use them in a shared or production environment. Review the provider qualification tests in tests/aws_qualification.rs before adapting an example to AWS.

Read and write files

let first = client
    .put_object("documents/readme.txt", b"first revision\n".to_vec())
    .await?;

client
    .put_object("documents/readme.txt", b"second revision\n".to_vec())
    .await?;

let current = client
    .get_object("documents/readme.txt")
    .await?
    .expect("current file");

let historical = client
    .get_object_at(first.id, "documents/readme.txt")
    .await?
    .expect("historical file");

assert_eq!(current.bytes, b"second revision\n");
assert_eq!(historical.bytes, b"first revision\n");

A standalone write and bulk ingestion both upload one whole content-addressed payload object for every distinct non-empty file body. Prolly does not combine small files, split large files, or persist byte extents or multipart state. Identical complete bodies reuse the same content-addressed object.

For safe retries, generate and persist one operation ID:

let operation = silo_s3_client::core::OperationId::new();
let receipt = client
    .put_object_with_operation("documents/readme.txt", bytes, operation)
    .await?;

After an ambiguous response, repeat the exact call with the same operation ID. The client reconciles an already-applied commit before fencing the writer.

Batch writes

Batching is the recommended bulk-write path. It uploads payloads while staging and publishes all tree changes with one branch compare-and-swap.

let mut commit = client
    .begin_commit()
    .message("import 2026-08-12")
    .checkpoint_every(256)
    .start()
    .await?;

let batch_id = commit.id();
commit.put_object("incoming/0001.json", first_body).await?;
commit.put_object("incoming/0002.json", second_body).await?;
commit.delete_object("incoming/obsolete.json")?;
commit.checkpoint().await?;

let receipt = commit.publish().await?;
println!("commit={} changed={}", receipt.id, receipt.changed_keys);

If a durable process stops after saving batch_id:

let mut commit = client.resume_commit(batch_id).await?;
commit.put_object("incoming/0003.json", third_body).await?;
let receipt = commit.publish().await?;

For a disposable job, add .ephemeral(). That removes checkpoint requests but cannot recover process-local staged metadata after a crash.

For a collection already in memory, put_objects creates durable batches for you. Payload uploads use bounded concurrency by default:

use silo_s3_client::PutObjectInput;

let receipts = client
    .put_objects(
        vec![PutObjectInput {
            key: "incoming/0004.json".into(),
            bytes: fourth_body,
            headers: Default::default(),
            user_metadata: Default::default(),
        }],
        1_000,
    )
    .await?;

For an unbounded or fallible source, pass a Stream so memory stays bounded by one checkpoint window:

use silo_s3_client::{BulkWriteOptions, PutObjectInput};

let receipts = client
    .put_object_stream(
        incoming_objects, // Stream<Item = silo_s3_client::Result<PutObjectInput>>
        BulkWriteOptions {
            batch_size: 10_000,
            concurrency: 32,
            checkpoint_every: 1_000,
        },
    )
    .await?;

Completed windows are durably checkpointed. On an input or per-object staging failure, the error's operation_id contains the resumable batch ID and its message identifies the failed object or completed checkpoint size. Dropping the future cancels in-flight work; the last completed remote checkpoint remains available to resume_commit.

List files and versions

let mut after: Option<String> = None;
loop {
    let (_snapshot, page, truncated) =
        client.list_objects("incoming/", after.as_deref(), 1_000).await?;
    after = page
        .last()
        .map(|item| String::from_utf8_lossy(&item.key).into_owned());
    for item in page {
        println!("{}", String::from_utf8_lossy(&item.key));
    }
    if !truncated {
        break;
    }
}

let (_head, versions) =
    client.list_object_versions("documents/readme.txt", 100).await?;

For large scans, prefer the snapshot-bound cursor or streaming APIs:

use futures_util::StreamExt;

let mut objects = client.stream_objects("incoming/", 1_000);
while let Some(object) = objects.next().await {
    consume(object?);
}

Paged and streamed listings honor the selected revision. For efficient historical traversal, detach from the branch handle that contains the commit; the clone retains that branch's derived node index as its lookup context:

let feature = client.checkout("feature").await?;
let historical = feature.checkout(commit_id).await?;

let first = historical.list_objects_page("incoming/", None, 1_000).await?;
assert_eq!(first.snapshot, commit_id);

let mut objects = historical.stream_objects("incoming/", 1_000);
while let Some(object) = objects.next().await {
    consume(object?);
}

list_objects_page returns an opaque continuation that seeks directly into the same immutable snapshot. A continuation is bound to its repository, branch, and prefix and cannot be reused for another query. Hold a retention pin on the returned snapshot if traversal may overlap garbage collection. Disable background_index_maintenance on a dedicated measurement client when provider telemetry must attribute only foreground reads; foreground listing itself never writes index state.

Branch and merge

use silo_s3_client::core::{MergePhase, MergePolicy};

let base = client.head().await?;
client.create_branch("feature", Some(base)).await?;

let feature = client.checkout("feature").await?;
feature
    .put_object("documents/feature.txt", b"feature\n".to_vec())
    .await?;

let mut merge = client
    .start_merge("feature", None, MergePolicy::Fail, "merge feature")
    .await?;

while merge.phase != MergePhase::ReadyToPublish {
    merge = client.advance_merge(&merge, 1_000).await?.cursor;
}

let page = client.merge_changes_page(&merge, None, 1_000).await?;
println!("changes={}", page.changes.len());
let receipt = client.publish_merge(&merge).await?;

Merge work is immutable and restartable. Persist the canonical MergeCursor returned by each bounded advance. Use merge_conflicts_page before publishing when the policy can produce conflicts.

Checkout branches, tags, and commits

checkout follows Git-like revision rules. An unqualified name resolves a branch before a tag. Fully qualified names remove ambiguity, and a CommitId creates a detached checkout:

use silo_s3_client::{CheckedOutRef, CheckoutRef};

let commit_id = client.head().await?;
client.create_tag("v1.0", commit_id).await?;

let feature = client.checkout("feature").await?;
assert_eq!(feature.branch(), Some("feature"));

let release = client.checkout("refs/tags/v1.0").await?;
assert_eq!(release.branch(), None);

let historical = client.checkout(commit_id).await?;
let object = historical.get_object("documents/readme.txt").await?;

// The typed form is useful when a branch and tag have the same name.
let release = client
    .checkout(CheckoutRef::Tag("v1.0".to_string()))
    .await?;
assert!(matches!(release.checked_out_ref(), CheckedOutRef::Tag { .. }));

Branch checkouts remain writable. Tag and commit checkouts are detached: head, get_object, head_object, listings, and log use their immutable snapshot, while mutation and branch-ref APIs return InvalidRevision.

History, diff, and recovery

History and diff cursors keep traversal state bounded:

use silo_s3_client::core::TraversalBudget;

let head = client.head().await?;
let log = client
    .log_bounded(head, None, 100, TraversalBudget::default())
    .await?;

let changes = client.diff_bounded(older, head, None, 1_000).await?;
for change in changes.changes {
    println!("{}", String::from_utf8_lossy(&change.key));
}

open_reflog captures a stable immutable journal snapshot. reset_branch moves a ref directly and records the move. start_restore instead creates new commits with fresh logical versions while reusing immutable payloads:

let expected = client.head().await?;
let mut restore = client
    .start_restore(older, expected, "restore known-good snapshot")
    .await?;

while !restore.complete {
    restore = client.advance_restore(&restore, 1_000).await?.cursor;
}

Persist the returned cursor after each page. Restores larger than the canonical commit limit are split into multiple atomic commits.

Integrity, repair, and backup verification

Metadata fsck validates commits, node packs, trees, logical versions, and payload metadata. Deep mode also downloads and hashes payload bytes:

let mut fsck = client.start_fsck(true).await?;
let fsck_job = fsck.job;
while fsck.phase != silo_s3_client::core::FsckPhase::Complete {
    fsck = client.advance_fsck(&fsck, 1_000).await?.cursor;
}
println!("verified {} commits", fsck.report.commits);
let mut cleanup = client.start_fsck_cleanup(fsck_job).await?;
while cleanup.phase != silo_s3_client::core::FsckCleanupPhase::Complete {
    cleanup = client.advance_fsck_cleanup(&cleanup, 1_000).await?.cursor;
}

Every page is checkpointed in the repository. A new process resumes with client.resume_fsck(fsck_job); checkpoint generations reject stale workers. Cleanup is permitted only after completion, is bounded, and can restart from start_fsck_cleanup after process loss.

Cross-provider repair is logical. It does not copy provider version IDs. It downloads verified source payloads, rebinds them at the destination, preserves logical metadata, and removes destination-only keys:

let source_snapshot = source.head().await?;
let destination_head = destination.head().await?;
let mut repair = destination
    .start_repair_from(
        &source,
        source_snapshot,
        destination_head,
        "repair from primary",
    )
    .await?;

loop {
    let page = destination.advance_repair_from(&source, &repair, 1_000).await?;
    repair = page.cursor;
    if page.complete { break; }
}

let mut verify = source
    .start_backup_verification(
        &destination,
        source_snapshot,
        repair.expected_head,
    )
    .await?;
while !verify.complete {
    verify = source
        .advance_backup_verification(&destination, &verify, 1_000)
        .await?
        .cursor;
}

The existing start_clone_from, start_fetch_from, and start_push_to methods remain snapshot-only aliases for compatibility. Use their history_ variants when commit topology matters:

let source_head = source.head().await?;
let expected_destination_head = destination.head().await?;
let mut transfer = destination
    .start_history_clone_from(
        &source,
        source_head,
        expected_destination_head,
    )
    .await?;

while !transfer.complete {
    transfer = destination
        .advance_history_transfer_from(&source, &transfer, 1_000)
        .await?
        .cursor;
    // Persist `transfer` here so the job can resume after a restart.
}

destination
    .publish_history_transfer(&transfer, "publish imported history")
    .await?;

History transfer walks the full source commit DAG parent-first, recreates each commit with destination-local payload bindings, preserves merge topology, and records the source-to-destination commit mapping. Commit IDs necessarily change because repository identity and provider bindings are different.

Retention pins are durable tag-backed roots:

client.create_retention_pin("quarter-close", head).await?;

Garbage collection

GC is restartable and bounded. Persist the returned cursor after every page. The grace period must be longer than the maximum time an upload, resumable commit, merge, repair, or history transfer may remain unpublished:

use silo_s3_client::core::GcPhase;

let two_hours_millis = 2 * 60 * 60 * 1_000;
let mut gc = client.start_gc(two_hours_millis).await?;

while gc.phase != GcPhase::Ready {
    gc = client.advance_gc(&gc, 1_000).await?.cursor;
    // Persist `gc` here.
}

while gc.phase != GcPhase::Complete {
    gc = match gc.phase {
        GcPhase::Ready | GcPhase::Sweeping => {
            client.sweep_gc(&gc, 1_000).await?.cursor
        }
        _ => client.advance_gc(&gc, 1_000).await?.cursor,
    };
    // Persist `gc` here.
}

Bulk ingest callers that use the journaled batch APIs can opt into journal-driven payload discovery:

let mut gc = client.start_gc_journaled(two_hours_millis).await?;

This mode reads one immutable creation-intent manifest per ingest window and does not list the payload namespace. Payloads from direct or pre-journal writers remain retained, so use start_gc while those writers remain active. A manifest contains whole-object paths, sizes, and checksums only—payload bytes are never packed or chunked.

The collector sweeps only immutable commit, direct-node, and whole-payload objects. It never sweeps mutable refs, derived indexes, publication journals, format markers, or administration data. Branch and tag updates during an epoch write dirty-root records before their CAS; sweep batches fence publication and catch up those roots before exact-version deletion. Retention pins are tags, so they are discovered and journaled by the same protocol.

Cache immutable nodes

The foyer-cache feature is enabled by default. Production deployments should use the cardinality-aware profile so persistent storage, metadata-node bounds, startup prewarming, and upper-level pinning are configured together:

use silo_s3_client::{Client, ProductionCacheProfile};

let profile = ProductionCacheProfile::new(
    "./prolly-node-cache",
    1_000_000, // expected live logical objects
);
let client = Client::builder()
    // supply the required AWS client, provider identity, signer, bucket, etc.
    .production_cache_profile(profile)
    .open()
    .await?;

let startup = client.startup_metrics();
let performance = client.performance_snapshot();
println!(
    "startup={}ms hit_ratio={:.3} metadata_amplification={:.3}x",
    startup.total_open_millis,
    performance.cache.hit_ratio(),
    performance.metadata_download_amplification(),
);

CacheSizingRecommendation::for_object_count exposes the selected memory, disk, location, and prewarm bounds without opening a client. Supplying the profile always opens persistent Foyer storage; --no-default-features callers must enable foyer-cache explicitly.

After stopping request traffic and dropping other client clones, call client.close_production_cache().await? to flush and close the cache opened by the production profile. This also stops the shared authority, branch-index, and telemetry maintenance tasks for that client.

For custom deployments, construct the cache directly:

use std::path::PathBuf;
use silo_s3_client::{FoyerNodeCache, FoyerNodeCacheConfig};

let cache = FoyerNodeCache::open(FoyerNodeCacheConfig {
    directory: PathBuf::from("./prolly-node-cache"),
    memory_capacity_bytes: 64 * 1024 * 1024,
    disk_capacity_bytes: 4 * 1024 * 1024 * 1024,
    // 32 MiB accommodates the repository format's 16 MiB hard node bound
    // plus Foyer's block index and entry envelope.
    disk_block_size_bytes: 32 * 1024 * 1024,
    memory_shards: 8,
})
.await?;

let client = Client::builder()
    // supply the same required provider and bucket settings
    .node_cache(cache.clone())
    .open()
    .await?;

// During graceful shutdown, release cache users before flushing Foyer.
drop(client);
cache.close().await?;

Cache keys include repository identity, tree format, and immutable CID. Cached bytes are verified before use. Persisted caches improve cold-start traversal but are never authoritative. max_entry_size_bytes() reports the largest node that fits the configured disk block; larger nodes are deliberately rejected from cache admission and continue to use the provider fallback.

Use prewarm_node_cache(snapshot) during startup to traverse both state trees. Use prewarm_node_cache_levels(snapshot, levels) when startup should load only the roots and shared upper paths instead of scanning every leaf. Use node_cache_snapshot() before and after to observe hits, misses, insertions, corruptions, coalesced waits, ranged fetches, requested/fetched/ avoided bytes, byte amplification, predictive prefetches, pinned nodes, and admission rejections. Take performance_snapshot() before and after a metadata-only operation and use delta_since plus metadata_download_amplification() to include commit, index, and control-object response bytes in the measured amplification.

OpenTelemetry

Enable the opentelemetry feature and supply the application-owned meter. The client records deltas on a bounded maintenance interval; the embedding service continues to own the SDK, exporter, resource attributes, and shutdown:

use std::{sync::Arc, time::Duration};
use opentelemetry::global;
use silo_s3_client::{Client, OpenTelemetryClientMetrics};

let telemetry = OpenTelemetryClientMetrics::new(global::meter("silo"));
let client = Client::builder()
    // supply the remaining required settings
    .telemetry(telemetry, Duration::from_secs(15))
    .open()
    .await?;

Metric names and alert thresholds should be defined by the application embedding the client and its production observability policy.

Performance model

• Every distinct logical payload is one complete immutable provider object. Exact duplicate bodies may reuse that complete object by content hash.
• Tiny-file batches upload those whole objects with bounded concurrency. Tree, append-only checkpoint, and publication requests are amortized across the batch; Prolly never combines file bodies or records payload extents. Each checkpoint window contains only keys changed since its predecessor.
• A current or historical read resolves a ref/commit/tree path and one payload.
• Warm immutable-node caches remove most repeated metadata reads.
• Branch creation inherits immutable derived-index roots from its source and is independent of snapshot size. Sparse diff and merge fetch compact commit descriptors and only the node ranges on their traversal frontier.
• Large commit deltas are external Prolly trees, keeping commit descriptors and restart metadata bounded independently of batch size.
• Callers that need provider-native multipart or resumable transfer use prepare_external_object_upload, upload the single final object with their provider transfer manager, then call stage_external_object_upload. Prolly persists no upload ID, part geometry, part ETags, chunks, or chunk manifest.
• Metadata node packs contain only Prolly index nodes. They never contain user object bodies and are not a payload storage or transfer format.
• Same-branch writers contend on one ref CAS; different branches publish independently.
• Concurrent independent callers can use ordered_publication_queue to apply bounded backpressure, prepare complete objects concurrently, and coalesce unique keys into one deterministic commit. Duplicate-key submissions cross a commit boundary to preserve version order. Acknowledgements are constant-size and are delivered only after the grouped ref CAS succeeds.

Measure with s3_operation_metrics on the provider and workload you will run. The repository enforces configured request-shape limits, but no universal latency or cost claim substitutes for AWS qualification.

Limitations

• The client must be the exclusive authority for its repository prefix.
• One file must fit the repository and provider single-object size limits.
• Prolly does not define or manage a chunked file representation. Its built-in upload path uses one PutObject up to the provider's 5 GiB single-PUT limit.
• Larger or resumable transfers must be completed by an external provider transfer manager and handed back as one whole object for verification and publication.
• The convenience put_stream path uses a bounded disk spool because it learns content identity at end-of-stream. The external-upload handoff avoids Prolly transfer state but requires the complete size and whole-object checksums.
• Concurrent writes to one branch can conflict; batch related changes.
• Concurrent GC closes publication admission through the durable repository coordinator, fences writer handles in other processes, checkpoints its epoch, and resumes after process restart. Writers bypassing the repository protocol remain unsupported.
• Provider-retained or legal-held versions that reject exact deletion are counted in GcReport::protected_versions/protected_bytes; they remain physically present while GC completes and publication admission reopens.
• Snapshot clone/fetch/push preserves only the selected logical state. The history_ variants preserve the source commit DAG, but not source commit IDs or reflog identity.
• “Millions or billions” requires provider quota, cache, latency, cost, throttling, and hot-branch qualification at the intended workload.

See the runnable rustfs_versioned_bucket example and the repository's development checks.